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METHOD AND DEVICE FOR ADJUSTING AN IMAGE SENSOR SYSTEM 
Background Information 

The present invention relates to a method and a device for adjusting at least one 
parameter of at least one image sensor of an image sensor system, the image 
sensor system including at least two image sensors. 

5 

Image sensor systems having at least two image sensors which essentially record 
the same scene are known. Such image sensor systems are also known as "stereo 
cameras." EP 1 028 387 A2, for example, describes a surroundings sensor device in 
a motor vehicle having a stereo camera. The stereo camera has two image sensors 

10 having overlapping viewing fields. It is proposed that the generated surroundings 
image data of both image sensors be used for detecting traffic signs. The detected 
traffic signs are displayed to the driver of the vehicle via a head-up display. The 
stereo camera is thus a component of a driver assistance system for displaying 
traffic signs in the motor vehicle. Driver assistance systems are systems in a motor 

1 5 vehicle which support the driver in road traffic by providing driver assistance 
functions. The reliable functioning of driver assistance systems and their 
components combined with high availability are the prerequisites for the use of these 
systems in a motor vehicle. EP 1 028 387 A2 provides no indication of a method or a 
device for achieving high availability of an image sensor system having at least two 

20 image sensors which essentially record the same scene. 

Advantages of the Invention 

The method and device for adjusting at least one parameter of at least one image 
sensor of an image sensor system, described below, having at least two image 

25 sensors recording essentially the same scene, have the advantage that the error 
tolerance of the image sensor system is enhanced. In a particularly advantageous 
manner, this generally contributes to high availability of the image sensor system, in 
particular of the above-described stereo camera. If a monocular error occurs, i.e., an 
error in only one of the image sensors of the image sensor system, the image sensor 

30 system does not necessarily fail. During continuous operation, the method and the 
device thus contribute to reducing downtimes and thus to increasing availability. 
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The method and the device described below are advantageous in motor vehicles in 
partiQular. Image sensor systems in motor vehicles are used in driver assistance 
systems and/or in safety-related systems. For example, the use of image sensor 
systems for recognizing the danger of an accident and preparing safety devices, 
5 such as air bags and/or seat belt tensioners, for the possible danger of an accident 
as a function of the recognized danger situation is planned. This permits controlled 
and more rapid deployment of these safety devices. Therefore, a high degree of 
availability is needed when image sensor systems are used in such safety-related 
systems. The above-described method and device contribute, in a particularly 

10 advantageous manner, to the high availability of an image sensor system in motor 
vehicles. It is also advantageous that the availability of the image sensor system is 
increased particularly in difficult driving situations such as rain. Such difficult driving 
situations entail a particularly high danger of an accident, and it is important that the 
image sensor system has high availability in such driving situations. However, even 

1 5 when image sensor systems are used in driver assistance systems, the method and 
the device described below advantageously contribute to increasing the availability 
of the driver assistance functions in difficult driving situations. This is advantageous 
because many driver assistance functions are designed particularly for use in difficult 
driving situations where they are needed by the driver. 

20 

When adjusting at least one lighting parameter, improved error tolerance of the 
lighting adjustment of the image sensors is advantageously achieved because in the 
case of an error in at least one image sensor, only the image sensors having error- 
free lighting information of at least one further image sensor are adjusted. It is 

25 particularly advantageous if the gain and/or offset and/or the integration time of the 
at least one image sensor is/are used as parameters. These parameters are 
particularly well suited for automatic lighting adjustment and the switchover of the 
lighting adjustment to at least one error-free image sensor described below. From 
the point of view of automatic adjustment and/or regulation and/or control of the 

30 lighting, a stereo camera has a particular property. The lighting sensor, i.e., the 
image sensor chip, on which the lighting adjustment depends, for example, is 
present in duplicate. It is particularly advantageous that in the case of a stereo 
camera both image sensors record essentially the same scene; and the lighting 
information obtained by both image sensors is thus essentially identical. This 
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redundancy may be used in the case of an error by using only the lighting 
inforrjiation from the error-free image sensor. 

Another advantage of the method and the device described below is that at least one 
5 image error and/or at least one hardware error is/are recognized as an error type. It 
is therefore possible not only to recognize image errors, but also errors in the 
hardware. The method and the device thus advantageously contribute to the high 
availability of an image sensor system even if an error occurs in the form of at least 
one hardware error in an Image sensor. 

10 

Particularly advantageous is a processing unit for generating at least one adjustment 
signal for at least one parameter of at least one image sensor of an image sensor 
system for performing all or at least the essential steps of the method described 
below. The advantages of such a processing unit are the above-described 
1 5 advantages of the method and the device for adjusting at least one parameter of at 
least one image sensor of an image sensor system. 

Particularly advantageous is a computer program having program code means to 
execute all steps or at least the essential steps of the above-described method if the 
20 program is run on a computer. The use of a computer program allows the method to 
be rapidly and inexpensively adapted, for example, by adapting to different image 
types. Alternatively or additionally, simple recording of additional error types is 
possible. 

25 Additional advantages result from the subsequent description of exemplary 

embodiments with reference to the figures and from the dependent patent claims. 

Drawing 

The present invention is elucidated below with reference to the embodiments 
30 illustrated in the drawing. The sole Figure 1 schematically shows the preferred 
exemplary embodiment. 
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Description of the Exemplary Embodiments 

A method and a device for adjusting at least one parameter of an image sensor of a 
stereo camera in a motor vehicle are described below. The stereo camera has two 
image sensors which essentially record the same scene. If an error occurs in one of 
5 the image sensors, at least one parameter of this image sensor is adjusted as a 
function of at least one measured value of the error-free image sensor of the image 
sensor system. In the preferred exemplary embodiment, the parameter is at least 
one lighting parameter of the image sensor. In the event of an error or malfunction of 
an image sensor of the stereo camera, the at least one lighting parameter is adjusted 
10 as a function of at least one measured value of the second image sensor, the 

measured value being a measure of the lighting of at least one part of the image of 
the second image sensor. 

Figure 1 schematically shows the preferred exemplary embodiment, which has a first 

15 image sensor 10, a second image sensor 20, and a processing unit 30 having 

different modules 32, 34, 36. 38. First image sensor 10 and second image sensor 20 
are situated in such a way that they record essentially the same scene. In the 
preferred exemplary embodiment, first image sensor 10 and second image sensor 
20 form a stereo camera. In a stereo camera, not only do the two image sensors 10, 

20 20 record essentially the same scene, but also the optical axes of the two image 

sensors 10, 20 are essentially parallel. In the preferred exemplary embodiment, both 
first image sensor 10 and second image sensor 20 are mounted at a mutual 
horizontal distance of approximately 0.2 m behind the windshield in the areas of the 
internal rear-view mirror of a motor vehicle. The two image sensors 10, 20 are 

25 oriented in such a way that their image detection area covers the surroundings of the 
motor vehicle in the direction of travel. In this exemplary embodiment, the lighting of 
the stereo camera pair is adjusted. If an error occurs in one of image sensors 10, 20, 
the lighting information for both image sensors 10, 20 together is obtained only from 
the image of the error-free image sensor 10, 20, while the erroneous lighting 

30 information of the defective image sensor 10, 20 is ignored. For example, if an error 
occurs in first image sensor 10, the lighting of both image sensors 10, 20 is adjusted 
as a function of the lighting information of second image sensor 20. For this purpose, 
the adjustment of image sensors 10, 20 is automatically switched over to the error- 
free image sensor 10, 20 for obtaining the lighting information. For this purpose, a 
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mechanism for automatic detection of error conditions is available in both Image 
sensprs 10, 20. The error conditions may be obtained either via monitoring functions 
of the hardware or via analysis of the images of image sensors 10, 20 for image 
errors. If an error is detected in one of image sensors 10, 20, the lighting information 
5 for both image sensors 1 0, 20 is obtained only on the basis of the error-free sensor. 
If both image sensors 10, 20 operate error-free, both image sensors 10, 20 are 
adjusted either individually or jointly. In the preferred exemplary embodiment, CMOS 
image sensors having a logarithmic lighting characteristic, which generate grayscale 
images having an eight-bit resolution, are used. First image sensor 10 transmits 

10 monitoring signals for error monitoring to module 32 via signal line 12. In a similar 
manner, monitoring signals are transmitted from second image sensor 20 to module 
34 via signal line 22. In the preferred exemplary embodiment, image signals and 
different hardware signals of image sensors 10, 20, are transmitted as monitoring 
signals. Errors in the two image sensors 10, 20 are detected in error detection 

15 modules 32, 34 from the monitoring signals transmitted via signal lines 12, 22. Error 
detection module 32 transmits error signals to image sensor switching module 36 via 
signal line 33. Enror signals from error detection module 34 to image sensor 
switching module 36 are similarly transmitted via signal line 35. Depending on 
whether an error condition has been detected in the two image sensors 10, 20, one 

20 of the three cases explained below occurs in image sensor switching module 36. In 
the first case, both image sensors 10, 20 operate error-free, and no error signals are 
transmitted to module 36 via signal lines 33, 35. Alternatively, it is possible to adjust 
both image sensors 10, 20, either jointly or individually. In the event of a joint 
adjustment, the lighting information from the two image sensors 10, 20 is averaged 

25 or combined in a joint grayscale histogram of both images. In the preferred 
exemplary embodiment, the lighting information is obtained in image sensor 
adjustment module 38 from image information transmitted to image sensor switching 
module 36 from first image sensor 10 via signal line 16 and from second image 
sensor 20 via signal line 26, and further to image sensor adjustment module 38 via 

30 signal line 37. In the case of individual adjustment, image sensors 10, 20 are either 
adjusted only with the help of the lighting information of one of the two image 
sensors 10, 20 or each image sensor 10, 20 is adjusted using its own lighting 
information. In the second case, an error is detected in one of the two image sensors 
10, 20. This error is transmitted from the corresponding error detection module 32, 
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34 to image sensor switching module 36 via one of signal lines 33, 35. In this case, 
only the lighting information of error-free image sensor 10, 20 is used in module 38 
for adjusting both image sensors 10, 20. For example, when there is an error in first 
image sensor 10, image information of second image sensor 20 Is transmitted via 
5 signal line 26 by image sensor switching module 36 and further to image sensor 
adjustment module 38. If an error is detected simultaneously in both image sensors 
10. 20, the third case is present. If it is a brief outlier, the instantaneous lighting 
information is ignored by image sensor adjustment module 38 and the lighting 
adjustment continues to run "blind." This means that the adjustment Is continued 

10 using the latest determined values until normal lighting conditions occur and/or until 
the error no longer exists in at least one image sensor 10, 20 and the lighting 
information of at least one image sensor 10, 20 may be used again for lighting 
adjustment. If the error continues in both image sensors, other measures must be 
taken. In the preferred exemplary embodiment, an alarm is given to the user, i.e., the 

1 5 driver of the motor vehicle, and/or the system is shut off. The actual lighting 

adjustment of Image sensors 10, 20 is computed in module 38. For this purpose, 
adjustment signals are generated from the Image information transmitted via signal 
line 37. The adjustment signals are transmitted to first Image sensor 10 via signal 
line 14 and to second image sensor 20 via signal line 24. In the preferred exemplary 

20 embodiment the adjustment signals are signals for adjusting at least one lighting 
parameter. The electrical gain and/or the offset and/or the integration time Is/are 
used as lighting parameters. In the preferred exemplary embodiment the lighting 
information is determined from the grayscale distribution of the transmitted image 
information. The mean of the grayscale values and/or a statistical value, in particular 

25 the median and/or the maximum and/or the minimum and/or the quantile of the 
grayscale histogram of the image information Is/are used as lighting Information. 

In the preferred exemplary embodiment, individual modules 32, 34, 36, 38 of 
processing unit 30 are implemented in a digital microprocessor which achieves the 
30 above-described functions of these modules 32, 34, 36, 38 as programs, 

subprograms, or program steps. In another variant, at least two microprocessors 
containing individual modules 32. 34, 36, 38 are provided. The signals are 
transmitted over signal lines 12. 14. 16, 22, 24, 26. 33, 35. 37 electrically or optically. 
Alternatively or additionally, wireless transmission is possible. In a further variant of 
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the preferred exemplary embodiment, both image sensors 10, 20 and processing 
unit 30 are combined to form a single unit. 

4) 

The error detection modules for the image sensors detect errors of a certain type. A 
5 distinction is made between two types of errors: hardware errors and image errors. 
Failures of hardware components of the image sensors are recognized by detection 
circuits. Hardware components include in particular the electronic components for 
converting the light signal into an electrical signal, for example, at least one image 
sensor chip and/or the analysis electronics including microprocessors and memories 

10 in the image sensor. Detection circuits are electronic circuits which generate an error 
signal when an error occurs in one of the circuits to be monitored. Hardware error 
types include errors in the communication between the image sensor chip and the 
analysis electronics. The communication is often implemented via a bus protocol 
using appropriate error detection. The failure of individual bit levels is another type of 

15 hardware failure. This failure is typically manifested in the entire image being 

represented by only 1 or only 0 level bits. The existence of such a bit level error is 
determined by appropriately checking a sufficient number of bits of an image. The 
second error type is image errors. The second error type (image error) includes in 
particular lighting errors and/or errors due to loss of contrast and/or highly distorted 

20 images. Lighting errors are recognized, for example, by analyzing the histogram of 
an image. In the case of overlighting or underlighting, an increase in the frequency of 
grayscale values in the upper or lower value range of the histogram occurs. Loss of 
contrast occurs, for example, when an image sensor is blocked. Such monocular 
errors are known in motor vehicles. If a stereo camera is operated behind the 

25 windshield and the windshield wiper is active at the same time, the windshield wiper 
briefly blocks one of the two image sensors in regular intervals. This results in 
temporary disturbance of the lighting adjustment of the image sensors. In particular, 
this causes the above-mentioned lighting errors and/or loss of contrast. Dynamic 
switch-over of the image sensor adjustment to the image sensor that is not blocked 

30 at the moment contributes to the prevention of such an undesirable condition. Highly 
distorted images occur, for example, at elevated operating temperatures and/or in 
the event of operating errors and/or communication errors of the image sensor chip. 
The image error category also includes errors due to outlier images. Outlier images 
are understood as occasionally occurring image disturbances and/or changes in 



lighting and/or blocking affecting only a few consecutive images. Innage disturbances 
occuf , for example, in the event of intermittent electrical contacts in the electrical 
circuits of the image sensors, while changes in lighting occur, for example, in the 
event of brief interferences due to reflections of strong light sources. In the case of 
5 image sensors in motor vehicles, interference due to reflections is caused, for 
example, by the headlights of other motor vehicles. Blocking in the image of an 
image sensor of a stereo camera occurs in a motor vehicle, for example, due to the 
above-mentioned case of the windshield wiper. Such outlier images are detected via 
predictive methods and/or by detecting the difference between instantaneous and 

10 previous images. If the difference between instantaneous and previous images 
exceeds a previously defined threshold, the instantaneous image is considered an 
outlier. The image error type also includes errors due to fuzzy images. Fuzzy images 
occur, for example, due to visibility impairments of the image sensors due to rain 
and/or dirt and/or water splash and/or defocusing. The occurrence of an error in one 

15 of the image sensors may be verified by comparing the right and left Images of the 
stereo cameras. The difference between the right and left images is usually slight. 
However, if an error occurs in only one of the image sensors, in general also the 
error between the two images is magnified. 

20 The above-described method and device for adjusting at least one image sensor of 
an image sensor system, where the image sensor system has at least two image 
sensors recording essentially the same scene, are not restricted to the adjustment of 
at least one lighting parameter. The above-described method and device are suitable 
in general for adjusting and/or controlling and/or regulating at least one parameter of 

25 at least one image sensor of an image sensor system, the image sensor system 

having at least two image sensors recording essentially the same scene. In addition 
to adjusting at least one lighting parameter, the adjustment and/or control and/or 
regulation of at least one focusing parameter is/are also possible. In today's image 
sensors, the focus of the image is adjusted automatically using autofocus. The focal 

30 distance of the image sensor lens is used, for example, as the focusing parameter. 
Alternatively or additionally, the aperture is adjusted as the parameter of at least one 
image sensor in other variants. Alternatively or additionally, in color reproduction 
image sensors, the white balance is adjusted as the parameter in another variant. 
Alternatively or additionally, the lighting sensitivity of the image sensor is adjusted in 
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another variant. For exannple, in a particular embodiment, the lighting sensitivity of 
the irnage sensor is adjusted by adjusting the break points in a characteristic curve 
having linear segments. In another variant, at least two different parameters of at 
least one image sensor are adjusted simultaneously using the above-described 
5 method and device. When at least one error of at least one error type occurs in at 
least one image sensor, the at least two different parameters of this image sensor 
are adjusted as a function of at least one measured value of at least one further 
image sensor of the image sensor system. 

10 An image sensor system having more than two image sensors recording essentially 
the same scene is used in one variant of the above-described method and device. 
Alternatively or additionally, at least one image sensor having a linear characteristic 
curve is used in a further variant. The image sensors used are not limited to black- 
and-white image sensors. Instead, the use of image sensors of different designs with 

15 respect to resolution and/or color depth and/or the lighting characteristic is possible. 
Alternatively or additionally, CCD image sensors of different designs with respect to 
resolution and/or color depth and/or the lighting characteristic may be used. In 
general, the above-described method, device, processing unit, and computer 
program are not limited to use in motor vehicles. 



